Flying camera and a system

ABSTRACT

There is provided a control device including an image display unit configured to acquire, from a flying body, an image captured by an imaging device provided in the flying body and to display the image, and a flight instruction generation unit configured to generate a flight instruction for the flying body based on content of an operation performed with respect to the image captured by the imaging device and displayed by the image display unit.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/595,978, filed Oct. 8, 2019, which is a continuation of USapplication Ser. No. 16/118,173, filed Aug. 30, 2018 (now U.S. Pat. No.10,469,757), which is a continuation of U.S. application Ser. No.15/656,870, filed Jul. 21, 2017, (now U.S. Pat. No. 10,104,297), whichis a continuation of U.S. application Ser. No. 14/227,182, filed Mar.27, 2014 (now U.S. Pat. No. 9,749,540), which claims the benefit ofPriority under 35 U.S.C. § 119 from Japanese Priority Patent ApplicationJP 2013-088456, filed Apr. 19, 2013. The entire contents of each of theabove applications are incorporated herein by reference.

BACKGROUND

The present disclosure relates to a control device, a control method,and a computer program.

A technology relating to a method for imaging photographs using a camerainstalled in a radio-controllable flying body has been disclosed (forexample, refer to JP 2006-27448). Using the camera installed in such aflying body, it is possible to image photographs from the sky or aposition in which a tripod is difficult to set. Imaging using a camerainstalled in a flying body brings various advantages in that costs canbe suppressed, and safe imaging, imaging at a low altitude or in anarrow place, imaging in proximity to a target, and the like arepossible in comparison to when a real aircraft or helicopter is used.

SUMMARY

In order to execute the imaging method described above, it is necessaryto manipulate the flying body using a proportional control system(Propo) or the like and then manipulate the camera. Thus, in order toexecute the imaging method described above, extensive training forfreely manipulating the flying body is necessary, and further, trainingfor manipulating the camera installed in the flying body is alsoindispensable.

Therefore, it is desirable to provide a novel and advanced controldevice, control method, and computer program that enable maneuvering ofa flying body in which a camera is installed through an instantaneousoperation.

According to an embodiment of the present disclosure, there is provideda control device including an image display unit configured to acquire,from a flying body, an image captured by an imaging device provided inthe flying body and to display the image, and a flight instructiongeneration unit configured to generate a flight instruction for theflying body based on content of an operation performed with respect tothe image captured by the imaging device and displayed by the imagedisplay unit.

According to an embodiment of the present disclosure, there is provideda control method including acquiring, from a flying body, an imagecaptured by an imaging device provided in the flying body and displayingthe image, and converting content of an operation performed with respectto the image captured by the imaging device and displayed in the step ofdisplaying into a flight instruction for the flying body.

According to an embodiment of the present disclosure, there is provideda computer program causing a computer to execute acquiring, from aflying body, an image captured by an imaging device provided in theflying body and displaying the image, and converting content of anoperation performed with respect to the image captured by the imagingdevice and displayed in the step of displaying into a flight instructionfor the flying body.

According to the embodiments of the present disclosure described above,it is possible to provide a novel and advanced control device, controlmethod, and computer program that enable maneuvering of a flying body inwhich a camera is installed through an instantaneous operation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. is an illustrative diagram showing an exterior example of animaging system according to an embodiment of the present disclosure;

FIG. 2 is an illustrative diagram showing an exterior example of animaging system according to an embodiment of the present disclosure;

FIG. 3 is an illustrative diagram showing an exterior example of animaging system according to an embodiment of the present disclosure;

FIG. 4 is an illustrative diagram showing an exterior example of aflying device 100 constituting the imaging system 10 according to theembodiment of the present disclosure;

FIG. 5 is an illustrative diagram showing an exterior example of aflying device 100 constituting the imaging system 10 according to theembodiment of the present disclosure;

FIG. 6 is an illustrative diagram showing a modified example of theflying device according to the embodiment of the present disclosure;

FIG. 7 is an illustrative diagram showing a modified example of theflying device according to the embodiment of the present disclosure;

FIG. 8 is an illustrative diagram showing a structure example of a strut111 that supports rotor covers 102 a to 102 d;

FIG. 9 is an illustrative diagram showing an appearance of the rotorcover 102 a turning along a slit part 112 of the strut 111;

FIG. 10 is an illustrative diagram showing an appearance of the rotorcover 102 a turning along a slit part 112 of the strut 111;

FIG. 11 is an illustrative diagram showing an appearance in which a usercontrols an operation of the flying device 100 using a controller 200;

FIG. 12 is a descriptive diagram showing a functional configurationexample of the flying device 100 and the controller 200 according to theembodiment of the present disclosure;

FIG. 13 is a flowchart showing an operation example of the imagingsystem 10 according to the embodiment of the present disclosure;

FIG. 14 is a descriptive diagram showing an example of informationdisplayed on a display unit 210 of the controller 200 according to theembodiment of the present disclosure;

FIG. 15 is a descriptive diagram showing an example of informationdisplayed on a display unit 210 of the controller 200 according to theembodiment of the present disclosure;

FIG. 16 is a descriptive diagram showing an example of informationdisplayed on a display unit 210 of the controller 200 according to theembodiment of the present disclosure;

FIG. 17 is an information display example of the display unit 210 when auser wants to set a position designated using tapping at the center ofan image;

FIG. 18 is an information display example of the display unit 210 when auser wants to set a position designated using tapping at the center ofan image;

FIG. 19 is an illustrative diagram showing a state in which a userexecutes a drag operation on the display unit 210;

FIG. 20 is an illustrative diagram showing a state after a user executesthe drag operation on the display unit 210;

FIG. 21 is an illustrative diagram showing a state in which a userexecutes a pinch operation on the display unit 210;

FIG. 22 is an illustrative diagram showing control of flight of theflying device 100 using image feedback;

FIG. 23 is an illustrative diagram showing an example of thecorrespondence relationship between a user operation performed withrespect to an image captured by an imaging device 101 and a movement ofthe flying device 100;

FIG. 24 is an illustrative diagram showing an example of thecorrespondence relationship between a user operation performed withrespect to an image captured by an imaging device 101 and a movement ofthe flying device 100;

FIG. 25 is an illustrative diagram showing an example of thecorrespondence relationship between a user operation performed withrespect to an image captured by an imaging device 101 and a movement ofthe flying device 100:

FIG. 26 is an illustrative diagram showing an example of thecorrespondence relationship between a user operation performed withrespect to an image captured by an imaging device 101 and a movement ofthe flying device 100;

FIG. 27 is an illustrative diagram showing an example of informationdisplayed on the display unit 210 of the controller 200;

FIG. 28 is an illustrative diagram showing an example of thecorrespondence relationship between a user operation performed withrespect to an image captured by the imaging device 101 and a movement ofthe flying device 100;

FIG. 29 is an illustrative diagram showing another example of thecorrespondence relationship between a user operation performed withrespect to an image captured by the imaging device 101 and a movement ofthe flying device 100;

FIG. 30 is an illustrative diagram showing a state of the flying device100 positioned inside a predetermined flight range 21 that has aninitial position 20 as the center thereof;

FIG. 31 is an illustrative diagram showing a state of the flying device100 attempting to fly beyond the predetermined flight range 21 that hasthe initial position 20 as the center thereof;

FIG. 32 is an illustrative diagram showing a state of the flying device100 attempting automatic landing;

FIG. 33 is an illustrative diagram showing an overview of an operationof the flying device 100;

FIG. 34 is a flowchart showing an operation example of the imagingsystem 10 according to the embodiment of the present disclosure;

FIG. 35 is an illustrative diagram showing a control example of aposition and an attitude of the flying device 100 based on imagerecognition;

FIG. 36 is an illustrative diagram showing a control example of aposition and an attitude of the flying device 100 based on imagerecognition;

FIG. 37 is an illustrative diagram showing a control example of aposition and an attitude of the flying device 100 based on imagerecognition; and

FIG. 38 is an illustrative diagram showing a control example of aposition and an attitude of the flying device 100 based on imagerecognition.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

Note that description will be provided in the following order:

<1. Embodiment of the present disclosure=>

[Exterior example of an imaging system]

[Functional configuration example of an imaging system]

[Operation example of a flying device and controller]

[2. Conclusion]

1. EMBODIMENT OF THE PRESENT DISCLOSURE

[Exterior Example of an Imaging System]

First, an exterior example of an imaging system according to anembodiment of the present disclosure will be described with reference tothe drawings. FIGS. 1 to 3 are illustrative diagrams showing an exteriorexample of the imaging system according to the embodiment of the presenttechnology. Hereinafter, the exterior example the imaging systemaccording to the embodiment of the present disclosure will be describedusing FIGS. 1 to 3.

The drawing of FIG. 1 show an illustrative diagram showing the exteriorexample of the imaging system 10 according to the embodiment of thepresent disclosure in a perspective view, and the drawing shown in FIG.2 is an illustrative diagram showing the exterior example of the imagingsystem 10 according to the embodiment of the present disclosure in aperspective view taken from the opposite side of the perspective viewshown in FIG. 1. In addition, the drawing shown in FIG. 3 is anillustrative diagram showing the exterior example of the imaging system10 according to the embodiment of the present disclosure in aperspective view taken from below.

The imaging system 10 according to the embodiment of the presentdisclosure is configured to include a flying device 100 and a controller200 that controls the flying device 100. The flying device 100 can flyby rotating rotors under the control of the controller 200. However,when the flying device 100 does not fly, the controller 200 isconfigured to be able to accommodate the rotors therein as shown inFIGS. 1 to 3.

The flying device 100 shown in FIGS. 1 to 3 can fly using four rotors.In FIGS. 1 to 3, rotor covers 102 a to 102 d protecting each of therotors are shown. The imaging system 10 according to the embodiment ofthe present disclosure can accommodate the rotor covers 102 a to 102 dinside the controller 200 as shown in FIGS. 1 to 3 when the flyingdevice 100 does not fly.

The imaging system 10 according to the embodiment of the presentdisclosure is configured such that the controller 200 is slidable in thedirection of A of FIG. 1. The imaging system 10 according to theembodiment of the present disclosure is configured such that thecontroller 200 is detached from the flying device 100 when thecontroller 200 is slid in the direction of A of FIG. 1.

The flying device 100 includes an imaging device 101 that captures stillimages or moving images. The imaging device 101 is constituted by alens, an image sensor such as a CCD (Charge Coupled Device) image sensoror a CMOS (Complementary Metal Oxide Semiconductor) image sensor, andthe like. The imaging device 101 included in the flying device 100executes capturing of still images or moving images under the control ofthe controller 200, and provides the captured images to the controller200.

The controller 200 controls a flight state of the flying device 100 andcapturing of images by the imaging device 101 included in the flyingdevice 100. The controller 200 performs control of the flight state andimages through wireless communication. As shown in FIG. 1, thecontroller 200 includes a display unit 210. A functional configurationof the controller 200 for controlling the flight state of the flyingdevice 100 and capturing of images by the imaging device 101 included inthe flying device 100 will be described later in detail.

The display unit 210 is configured as, for example, a flat displaydevice such as a liquid crystal display device, or an organic EL displaydevice, and can display images captured by the imaging device 101 andinformation for controlling operations of the flying device 100. Thedisplay unit 210 includes a touch panel, and a user can perform a directoperation with respect to information displayed on the display unit 210by touching the display unit 210 with a finger, or the like.

So far, the exterior example of the imaging system 10 according to theembodiment of the present disclosure has been described using FIGS. 1 to3. Next, an exterior example of the flying device 100 will be describedin more detail.

FIG. 4 is an illustrative diagram showing an exterior example of theflying device 100 constituting the imaging system 10 according to theembodiment of the present disclosure. Hereinafter, the exterior exampleof the flying device 100 will be described using FIG. 4.

FIG. 4 shows a state of the imaging system 10 according to theembodiment of the present disclosure shown in FIG. 1 from which thecontroller 200 is detached. FIG. 4 shows spring parts 103 a to 103 d inaddition to the imaging device 101 and the rotor covers 102 a to 102 dshown in FIGS. 1 to 3. Each of the spring parts 103 a to 103 d is woundaround struts in the four corners of the flying device 100, and causesthe rotor covers 102 a to 102 d to rotate using the struts as axes.

The drawing of FIG. 4 shows a state in which the rotor covers 102 a to102 d are accommodated inside the flying device 100 by a user. Theimaging system 10 according to the embodiment of the present disclosurehas the form as shown in FIGS. 1 to 3 when the controller 200 covers theflying device 100 in the state of the form of the flying device 100shown in FIG. 4. When the controller 200 covers the flying device 100,the controller 200 suppresses turning of the rotor covers 102 a to 102d.

FIG. 4 shows a strut 105 provided at the center of the flying device100, a rotor 106 provided in the strut 105, and a plurality of ventholes 107. The rotor 106 performs rotational motions using the strut 105as an axis under the control of the controller 200. In addition, thevent holes 107 can be provided to correspond to a range of therotational motions of the rotor 106. Since the flying device 100 has therotor 106, a lift force can increase due to the rotational motions ofthe rotor 106 even when it is difficult to obtain a sufficient liftforce only from rotational motions of the rotors protected by the rotorcovers 102 a to 102 d.

FIG. 5 is an illustrative diagram showing an exterior example of theflying device 100 constituting the imaging system 10 according to theembodiment of the present disclosure. The drawing of FIG. 5 shows thestate in which the controller 200 is detached by sliding the controller200 in the direction of A of FIG. 1 from the state shown in FIG. 1. Whenthe controller 200 is slid in the direction of A of FIG. 1 from thestate shown in FIG. 1 and then the controller 200 is detached, the rotorcovers 102 a to 102 d of which turning has been suppressed by thecontroller 200 turn as shown in FIG. 5 due to actions of the springparts 103 a to 103 d.

FIG. 5 shows the rotors 104 a to 104 d protected by the rotor covers 102a to 102 d. The rotors 104 a to 104 d cause the flying device 100 torise by independently performing rotational motions according to controlof the controller 200.

In the flying device 100, the rotors 104 a to 104 d protected by therotor covers 102 a to 102 d are provided with a predetermined differencein level as shown in FIGS. 4 and S. By providing the rotors 104 a to 104d with the predetermined difference in level in this manner, the flyingdevice 100 can compactly accommodate the rotors 104 a to 104 d when therotors should be accommodated therein.

Since an existing flying device has a unit that outputs a thrust forcefixed thereto, portability thereof is impaired. Since the flying device100 constituting the imaging system 10 according to the embodiment ofthe present disclosure is configured to be able to accommodate therotors 104 a to 104 d therein as shown in FIGS. 4 and 5, portabilitythereof can be remarkably improved.

In addition, since the rotor covers 102 a to 102 d turn such that therotor covers pop up out of the flying device 100 in the state in whichthe controller 200 is detached from the flying device 100, the distancesbetween the center of gravity and the centers of the rotors 104 a to 104d can be lengthened in comparison to the state in which the rotor covers102 a to 102 d are accommodated, and an attitude during flight of theflying device 100 can be stabilized.

Herein, the principle of flight of the flying device 100 will bedescribed. With respect to the flying device 100, operations oflevitation, movement, stand-still, landing, and the like using therotors 104 a to 104 d of which rotation speeds are independentlycontrollable are possible. The rotation direction of the rotors 104 aand 104 c is exactly opposite to the rotation direction of the rotors104 b and 104 d, and if all rotors are rotated at a uniform speed bysetting the rotation direction of the rotors 104 a and 104 c to beexactly opposite to the rotation direction of the rotors 104 b and 104d, the flying device 100 ascends or descends.

In addition, for example, when a rotation speed of the rotors 104 a and104 b is lower than a rotation speed of the rotors 104 c and 104 d inthe state in which the flying device 100 is ascending, the flying device100 can move in the direction of the rotors 104 a and 104 b in theascending state. In addition, for example, when a rotation speed of therotors 104 a and 104 c is lower than a rotation speed of the rotors 104b and 104 d in the state in which the flying device 100 is ascending,the flying device 100 can rotate clockwise or counterclockwise in thehorizontal direction in the ascending state.

In this manner, by appropriately changing rotation speeds of the rotors104 a to 104 d, operations of the flying device 100 including lifting,horizontal movement, stand-still, landing and the like can be performed.In addition, in the present embodiment, such control of movements of theflying device 100 using changes of the rotation speed of the rotors 104a to 104 d can be performed by an operation with respect to imagescaptured by the imaging device 101, rather than a manipulation withrespect to the flying device 100.

In this manner, a position and attitude of the flying device 100 can befreely changed by controlling the rotation speed of the rotors 104 a to104 d, but it is desirable to ascertain a position of the device itselfand a relative position with respect to an imaging target in anenvironment in order to perform imaging using the imaging device 101, inaddition to control of the rotation speeds of the rotors 104 a to 104 d.

Methods for ascertaining a position of the device itself and a relativeposition with respect to an imaging target in an environment include,for example, a method in which an acceleration sensor, a gyro sensor, orother inertial sensor is used, a method in which a position of thedevice is estimated by itself based on movement amounts of a pluralityof target points by recognizing a feature point or an object in anenvironment using an image sensor and the like.

For example, a current position and attitude can be obtained from achange amount of a position and an attitude by measuring an accelerationand angular velocity of the device itself using an inertial sensor suchas an acceleration sensor or a gyro sensor. Furthermore, a position andattitude of the device itself can be obtained while an error caused byan integration of the change amount is corrected by measuring theabsolute amount of a direction, an altitude, and the like using apneumatic sensor.

In addition, for example, a position of the device itself can beestimated based on a movement amount of a plurality of target points byrecognizing a feature point or an object in an environment using animage sensor, or the like. This technique is called SLAM (SimultaneousLocalization And Mapping). By using SLAM in accordance with a movementamount obtained from the inertial sensor described above when SLAMshould be used, position recognition accuracy can be increased.

With such control, the flying device 100 can properly ascertain itsposition under an environment in which the flying device attempts toperform imaging, and can automatically perform a movement to a properposition and stand-still with feedback control.

Note that, although the case in which four rotors are provided has beendescribed in the above-described example, except for the rotor 106provided at the center of the flying device 100, the number of rotors isnot limited to the example. For example, the flying device may haveeight rotors.

FIGS. 6 and 7 are illustrative diagrams showing a modified example ofthe flying device according to the embodiment of the present disclosure.FIGS. 6 and 7 are illustrative diagrams showing an exterior example ofthe flying device 100 that has eight rotors. In FIGS. 6 and 7, theflying device 100 with rotor covers 102 a′ to 102 h′ is shown. Theflying device 100 having eight rotors can also be provided as a modifiedexample of the embodiment of the present disclosure as shown in FIGS. 6and 7.

In the example described above, although the action of the spring parts103 a to 103 d causes the rotor covers 102 a to 102 d to turn, thepresent disclosure is not limited to the example. For example, a slitmay be provided in each strut in which the rotor covers 102 a to 102 dare provided so that the rotor covers 102 a to 102 d turn along theslit.

FIG. 8 is an illustrative diagram showing a structure example of a strut111 that supports the rotor covers 102 a to 102 d. The strut 111 shownin FIG. 8 is provided with a slit part 112 that causes the rotor covers102 a to 102 d to turn around the strut 111. By causing the rotor covers102 a to 102 d to turn along the slit part 112, when the rotor covers102 a to 102 d open toward the outside of the flying device 100, all ofthe rotor covers 102 a to 102 d can be positioned at the same height.

FIGS. 9 and 10 are illustrative diagrams showing an appearance of therotor cover 102 a turning along the slit part 112 of the strut 111.

Hereinabove, the exterior example of the flying device 100 has beendescribed using FIGS. 4 and 5. Next, the controller 200 that controlsoperations of the flying device 100 will be described.

FIG. 11 is an illustrative diagram showing an example of an appearancein which a user controls an operation of the flying device 100 using thecontroller 200. When the controller 200 is detached from the flyingdevice 100, the controller functions as a device that can remotelymanipulate the flying device 100 as shown in FIG. 11. When the user usesthe controller 200, takeoff, flight, and landing of the flying device100 can be controlled.

In addition, the imaging system 10 according to the embodiment of thepresent disclosure can transmit an image captured by the imaging device101 provided in the flying device 100 to the controller 200 in realtime, and can control operations of the flying device 100 by receivingan operation with respect to an image captured by the imaging device 101and displayed on the display unit 210 of the controller 200. In otherwords, the user can control operations of the flying device 100 with anoperation with respect to an image captured by the imaging device 101,rather than manipulation with respect to the flying device 100.

Hereinabove, the exterior example of the imaging system according to theembodiment of the present disclosure has been described. Next, afunctional configuration example of the imaging system according to theembodiment of the present disclosure will be described.

[Functional Configuration Example of the Imaging System]

FIG. 12 is a descriptive diagram showing a functional configurationexample of the flying device 100 and the controller 200 according to theembodiment of the present disclosure. Hereinbelow, the functionalconfiguration example of the flying device 100 and the controller 200according to the embodiment of the present disclosure will be describedusing FIG. 12.

As shown in FIG. 12, the flying device 100 according to the embodimentof the present disclosure is configured to include the imaging device101, the rotors 104 a to 104 d, motors 108 a to 108 d, a control unit110, a communication unit 120, a sensor unit 130, a position informationacquisition unit 132, an alert issuing unit 140, and a battery 150.

In addition, as shown in FIG. 12, the controller 200 according to theembodiment of the present disclosure is configured to include thedisplay unit 210, a communication unit 220, a control unit 230, aposition information acquisition unit 240, a battery 242, and a powersupply unit 244. In addition, the control unit 230 is configured toinclude a flight instruction generation unit 232 and a display controlunit 234.

The control unit 110 controls operations of the flying device 100. Forexample, the control unit 110 can control adjustment of rotation speedsof the rotors 104 a to 104 d according to adjustment of rotation speedsof the motors 108 a to 108 d, an imaging process performed by theimaging device 101, a transmission and reception process of informationwith another device (for example, the controller 200) via thecommunication unit 120, an alert issuing process performed by the alertissuing unit 140, and the like.

The imaging device 101 includes a lens, an image sensor such as a CCDimage sensor or a CMOS image sensor, and the like as described above.The imaging device 101 included in the flying device 100 executesimaging of still images or moving images under the control of thecontroller 200. Images captured by the imaging device 101 aretransmitted from the communication unit 120 to the controller 200.

The rotors 104 a to 104 d cause the flying device 100 to fly bygenerating a lift force from rotation. Rotation of the rotors 104 a to104 d is caused by rotation of the motors 108 a to 108 d. The motors 108a to 108 d cause the rotors 104 a to 104 d to rotate. The rotation ofthe motors 108 a to 108 d can be controlled by the control unit 110.

The communication unit 120 performs transmission and reception processesof information with the controller 200 through wireless communication.The flying device 100 transmits images captured by the imaging device101 from the communication unit 120 to the controller 200. In addition,the flying device 100 receives instructions relating to flight from thecontroller 200 using the communication unit 120.

The sensor unit 130 is a group of devices that acquire a state of theflying device 100, and can include, for example, an acceleration sensor,a gym sensor, an ultrasonic sensor, a pneumatic sensor, and the like.The sensor unit 130 can convert an acquired state of the flying device100 into a predetermined signal, and provide the signal to the controlunit 110 if necessary. The position information acquisition unit 132acquires information of a current position of the flying device 100using, for example, the GPS (Global Positioning System) or the like. Theposition information acquisition unit 132 can provide the acquiredinformation of the current position of the flying device 100 to thecontrol unit 110 if necessary.

The alert issuing unit 140 generates an alert using a sound, light, orthe like based on control of the control unit 110 when the flying device100 attempts to fly over a pre-set flight range.

The battery 150 stores electric power for operating the flying device100. The battery 150 may be a primary battery that can only performdischarge, or may be a secondary battery that can also perform charge.When the battery 150 is a secondary battery, and the flying device 100is integrated with the controller 200 as shown in, for example, FIG. 1or the like, and the battery 150 can receive supply of electric powerfrom the controller 200.

The display unit 210 includes a flat display device, for example, aliquid crystal display device, an organic EL display device, or the likeas described above. The display unit 210 can display, for example,images captured by the imaging device 101 or information for controllingoperations of the flying device 100. The display unit 210 is providedwith a touch panel, and thus a user can perform a direct operation withrespect to the information displayed on the display unit 210 by touchingthe display unit 210 with his or her finger, or the like.

The communication unit 220 transmits and receives information with theflying device 100 through wireless communication. The controller 200receives images captured by the imaging device 101 from the flyingdevice 100 using the communication unit 220. In addition, the controller200 transmits instructions relating to flight of the flying device 100to the flying device 100 from the communication unit 220. Commandsrelating to flight of the flying device 100 can be generated by thecontrol unit 230.

The control unit 230 controls operations of the controller 200. Forexample, the control unit 230 can control a display process of text,figures, images, and other information on the display unit 210, atransmission and reception process of information with another device(for example, the flying device 100) via the communication unit 220, apower supply process performed by the power supply unit 244 with respectto the flying device 100, and the like.

The flight instruction generation unit 232 generates instructionsrelating to flight of the flying device 100. In the present embodiment,the flight instruction generation unit 232 generates the instructionsrelating to flight of the flying device 100 based on an operation withrespect to images captured by the imaging device 101. As the flightinstruction generation unit 232 generates the instructions relating toflight of the flying device 100 based on the operation with respect toimages captured by the imaging device 101, the controller 200 enables auser who is not skilled at maneuvering the flying device 100 to easilymaneuver the flying device 100. In addition, as the flight instructiongeneration unit 232 generates the instructions relating to flight of theflying device 100 based on the operation with respect to images capturedby the imaging device 101, flight instructions for causing the flyingdevice 100 to fly in a formation desired by the user can be generated.Note that a specific example of a process of generating an instructionrelating to flight of the flying device 100 by the flight instructiongeneration unit 232 will be described later.

The display control unit 234 controls display of text, figures, images,and other information on the display unit 210. Display of text, figures,symbols, images, and other information on the display unit 210 indrawings to be referred to in later description is assumed to becontrolled by the display control unit 234.

The position information acquisition unit 240 acquires information of acurrent position of the controller 200 using, for example, the GPS(Global Positioning System) or the like. The position informationacquisition unit 240 can provide the acquired information of the currentposition of the controller 200 to the control unit 230 if necessary.

The battery 242 stores electric power for operating the controller 200.The battery 242 may be a primary battery that can only performdischarge, or may be a secondary battery that can also perform charge.When the flying device 100 is integrated with the controller 200 asshown in, for example, FIG. 1 or the like, the battery 242 can executesupply of electric power to the flying device 100 via the power supplyunit 244. When the flying device 100 is integrated with the controller200 as shown in, for example, FIG. 1 or the like, the power supply unit244 supplies electric power stored in the battery 242 to the flyingdevice 100 under the control of the control unit 230.

The flying device 100 and the controller 200 constituting the imagingsystem 10 according to the embodiment of the present disclosure have theconfiguration as shown in FIG. 12, and thus enable maneuvering of theflying device 100 based on the operation of images which are captured bythe imaging device 101 and displayed on the display unit 210 of thecontroller 200. In other words, the user can control operations of theflying device 100 with the operation of the images captured by theimaging device 101, rather than manipulation of the flying device 100.

Information can be transmitted and received between the flying device100 and the controller 200 through wireless communication using afrequency band of, for example, 2.4 GHz, 5 GHz, or the like based on thestandard of IEEE 802.11, IEEE 802.15.1, or the like.

Hereinabove, the functional configuration example of the flying device100 and the controller 200 according to the embodiment of the presentdisclosure has been described using FIG. 12. Next, an operation exampleof the imaging system 10 according to the embodiment of the presentdisclosure will be described.

[Operation Example of an Imaging System]

FIG. 13 is a flowchart showing an operation example of the imagingsystem 10 according to the embodiment of the present disclosure.Particularly, the drawing is of a flowchart showing an operation exampleof the controller 200 constituting the imaging system 10. The drawingshown in FIG. 13 is of an example of an operation performed when a userdetaches the controller 200 from the flying device 100 and maneuvers theflying device 100 using the controller 200. Hereinbelow, the operationexample of the imaging system 10 according to the embodiment of thepresent disclosure will be described using FIG. 13.

First, the controller 200 transmits a takeoff instruction based on usermanipulation to the flying device 100 placed in a stand-still state on atable or a palm of the user (Step S101). When the flight instructiongeneration unit 232 detects that a predetermined manipulationcorresponding to the takeoff instruction has been performed on thedisplay unit 210 with a touch panel, the controller 200 causes theflight instruction generation unit 232 to generate the takeoffinstruction of the flying device 100 and transmits the generated takeoffinstruction to the flying device 100 through wireless communication.

When the flying device 100 receives the takeoff instruction from thecontroller 200, the control unit 110 causes the motors 108 a to 108 d torotate. Then, the flying device 100 ascends with a lift power generatedfrom rotation of the rotors 104 a to 104 d caused by the rotation of themotors 108 a to 108 d.

Then, the controller 200 receives an image captured by the imagingdevice 101 provided in the flying device 100 through wirelesscommunication, and then causes the image to be displayed on the displayunit 210 (Step S102). While the image captured by the imaging device 101is displayed on the display unit 210, the controller 200 stands by untilan operation with respect to the image displayed on the display unit 210is performed (Step S103).

For example, when the user performs an operation with respect to theimage displayed on the display unit 210 by touching the display unit210, or the like, the controller 200 then detects the content of theoperation with respect to the image displayed on the display unit 210 inthe flight instruction generation unit 232 (Step S104). When the contentof the operation with respect to the image displayed on the display unit210 is detected, the controller 200 then converts the content of theoperation into a flight instruction of the flying device 100 in theflight instruction generation unit 232 (Step S105).

Although a specific example will be described later in detail, forexample, when the content of the operation detected in Step S104 is thata subject that the user designates is to be positioned at the center ofthe image captured by the imaging device 101, the flight instructiongeneration unit 232 converts the operation executed by the user into theflight instruction of the flying device 100.

When the content of the operation is converted into the flightinstruction of the flying device 100 in Step S105 described above, thecontroller 200 then transmits the flight instruction obtained in theStep S105 described above to the flying device 100 through wirelesscommunication (Step S106). The flying device 100 controls rotation ofthe motors 108 a to 108 d using the control unit 110 based on the flightinstruction transmitted from the controller 200.

The imaging system 10 according to the embodiment of the presentdisclosure enables maneuvering of the flying device 100 based on theoperation with respect to the image captured by the imaging device 101and displayed on the display unit 210 of the controller 200 as thecontroller 200 is operated as described above. In other words, the usercan control operations of the flying device 100 with the operation withrespect to the image captured by the imaging device 101, rather thanmanipulation with respect to the flying device 100.

When the user views the image captured by the imaging device 101 anddisplayed on the display unit 210 and comes up with a desired formation,the user transmits an imaging instruction to the imaging device 101using the controller 200. The flying device 100 transfers the imaginginstruction from the control unit 110 to the imaging device 101 based onthe imaging instruction transmitted from the controller 200. The imagingdevice 101 executes an imaging process based on the imaging instructiontransferred from the control unit 110. Then, the flying device 100transmits an image obtained from the imaging process of the imagingdevice 101 to the controller 200.

Hereinabove, the operation example of the imaging system 10 according tothe embodiment of the present disclosure has been described. Next, anexample of information displayed on the display unit 210 of thecontroller 200 constituting the imaging system 10 according to theembodiment of the present disclosure will be described.

FIG. 14 is a descriptive diagram showing an example of informationdisplayed on the display unit 210 of the controller 200 according to theembodiment of the present disclosure. Hereinbelow, the example ofinformation displayed on the display unit 210 of the controller 200 willbe described using FIG. 14.

In FIG. 14, a state in which a takeoff and landing button v11 and animaging button v12 are displayed is shown superimposed on an imagecaptured by the imaging device 101. The takeoff and landing button v11is a button for causing the flying device 100 to take off or land. Auser can cause the flying device 100 to take off or land by touching thetakeoff and landing button vi. When the rotors 104 a to 104 d do notrotate, the takeoff and landing button v11 functions as a button forcausing the flying device 100 to take off, and when at least one of therotors 104 a to 104 d rotates, the takeoff and landing button v11functions as a button for causing the flying device 100 to land.

The imaging button v12 is a button for causing the imaging device 101 toexecute an imaging process. The user can cause the imaging device 101 tocapture still images or moving images by touching the imaging button v12to cause the imaging device 101 to execute the imaging process.

In addition, in FIG. 14, a state in which an object frame v13, a targetposition v14, and a moving arrow v15 are displayed is shown superimposedon the image captured by the imaging device 101. The object frame v13has a function of indicating a region designated by an operation of theuser. The target position v14 has a function of showing a targetposition of the region indicated by the object frame v13 in the capturedimage. The moving arrow v15 has a function of indicating a linear routeof the object frame v13 to reach the target position v14.

When the user performs an operation to cause the region surrounded bythe object frame v13 to reach the target position v14 on the imagecaptured by the imaging device 101, the flying device 100 controls itsown position and attitude such that the region surrounded by the objectframe v13 reaches the target position v14. In addition, the controller200 notifies the user of the fact that the flying device 100 is changingthe position and the attitude by causing the moving arrow v15 to bedisplayed on the display unit 210 until the region surrounded by theobject frame v13 reaches the target position v14.

A specific information display example of the display unit 210 will bedescribed in more detail. First, an information display example of thedisplay unit 210 when a user wants to set his or her designated positionat the center of an image will be described.

FIGS. 15 and 16 are illustrative diagrams showing an example ofinformation displayed on the display unit 210 of the controller 200according to the embodiment of the present disclosure. The drawingsshown in FIGS. 15 and 16 are of an information display example of thedisplay unit 210 when the user wants to set his or her designatedposition at the center of the image. Hereinafter, the example ofinformation displayed on the display unit 210 of the controller 200 willbe described using FIGS. 15 and 16.

When the user wants to set his or her designated position at the centerof the image, the controller 200 causes the user to perform, forexample, an operation of tapping the designated position once. FIG. 15shows a state in which the user executes the operation of tapping agiven spot of the display unit 210 once. When the user taps the givenspot of the display unit 210 once, the controller 200 causes the objectframe v13 and the target position v14 to be displayed on the displayunit 210. The display control unit 234 executes the display process ofthe display unit 210.

When the user taps the given spot of the display unit 210 once and theflight instruction generation unit 232 detects the tapping, the flightinstruction generation unit 232 generates a flight instruction thatcauses the flying device 100 to fly so that the region surrounded by theobject frame v13 is positioned in the target position v14. In addition,when the user taps the given spot of the display unit 210 once and theflight instruction generation unit 232 detects the tapping, the displaycontrol unit 234 executes display control such that the moving arrow v15connecting the object frame v13 and the target position v14 is displayedon the display unit 210.

FIG. 16 shows a state after the user executes the operation of tappingthe given spot of the display unit 210 once. As shown in FIG. 16, whenthe user executes the operation of tapping the given spot of the displayunit 210 once, the display control unit 234 causes the moving arrow v15connecting the object frame v13 and the target position v14 to bedisplayed on the display unit 210. The moving arrow v15 is displayed onthe display unit 210 until the object frame v13 reaches the targetposition v14.

FIGS. 17 and 18 show an information display example of the display unit210 when the user wants to set a position designated using tapping atthe center of an image. As shown in FIG. 17, when the user taps a givenspot of the display unit 210 once, the object frame v13, the targetposition v14, and the moving arrow v15 are displayed on the display unit210. Then, when the flying device 100 reaches a desired position, acentering process in which the position designated by the user usingtapping is set at the center of the image is executed as shown in FIG.18.

When the user wants to set his or her designated position in a desiredposition, the controller 200 causes the user to perform, for example, anoperation of dragging the designated position on the display unit 210.FIG. 19 is an illustrative diagram showing a state in which a userexecutes a drag operation on the display unit 210. When the userexecutes the drag operation of touching a given spot of the display unit210 and moving his or her finger on the display unit 210 while touchingthe spot, the controller 200 causes the object frame v13 and the targetposition v14 to be displayed on the display unit 210. The displaycontrol unit 234 executes the display process of the display unit 210.

When the flight instruction generation unit 232 detects the dragoperation of the user, the flight instruction generation unit 232generates a flight instruction that causes the flying device 100 to flyso that the region surrounded by the object frame v13 is positioned inthe target position v14. In addition, when the flight instructiongeneration unit 232 detects the drag operation of the user, the displaycontrol unit 234 executes display control such that the moving arrow v15connecting the object frame v13 and the target position v14 is displayedon the display unit 210.

FIG. 20 is an illustrative diagram showing a state after a user executesthe drag operation on the display unit 210. As shown in FIG. 20, whenthe user executes the drag operation on the display unit 210, thedisplay control unit 234 causes the moving arrow v15 connecting theobject frame v13 and the target position v14 to be displayed the displayunit 210. The moving arrow v15 is displayed on the display unit 210until the object frame v13 reaches the target position v14.

When the user wants to change magnification of an image, the controller200 causes the user to perform, for example, an operation of spreadingor closing two fingers on the display unit 210, that is, a so-calledpinch operation. FIG. 21 is an illustrative diagram showing a state inwhich the user executes the pinch operation on the display unit 210.When the user executes the operation of touching a given spot of thedisplay unit 210 with two fingers and spreading or closing the fingerson the display unit 210 while bringing the fingers into contact with thespot in the so-called pinch operation, the controller 200 causes theobject frame v13 and the target position v14 to be displayed on thedisplay unit 210. The display control unit 234 executes the displayprocess of the display unit 210.

When the user has executed the pinch operation on the display unit 210,the display control unit 234 decides the object frame v13 having themiddle point of the two fingers as the center thereof and causes theobject frame to be displayed on the display unit 210. In addition, whenthe user has executed the pinch operation on the display unit 210, thedisplay control unit 234 decides the target position v14 based on amovement amount to a position away from the positions with which theuser's fingers come into contact and causes the target position to bedisplayed on the display unit 210.

As described above, the controller 200 that has detected the useroperation performed on the image captured by the imaging device 101 anddisplayed on the display unit 210 causes a flight instructioncorresponding to the user operation to be generated and transmitted tothe flying device 100. Here, the generation of the flight instruction bythe controller 200 will be described in more detail.

Flight of the flying device 100 is controlled by the controller 200 bygiving image feedback so that an image captured by the imaging device101 is in a state desired by a user. FIG. 22 is an illustrative diagramshowing control of flight of the flying device 100 using image feedback.

When a user touches the display unit 210 with a touch panel as shown onthe left side of FIG. 22, the flight instruction generation unit 232detects the moment, and registers, as a reference image 251, a region ofa predetermined size having the position that the user touches with hisor her finger as the center. Since a target position is at the center ofan image during the centering process described above caused by atapping operation on a screen, the flight instruction generation unit232 computes an image movement amount from the position that the usertouches with his or her finger to the target position. Then, the flightinstruction generation unit 232 computes a flight control command basedon the computed image movement amount, and then transmits the command tothe flying device 100.

In the next frame, an image captured by the imaging device 101 ischanged due to a change of an attitude of the flying device 100. Thus,the flight instruction generation unit 232 performs image searching inthe periphery of the position of the reference image 251 in the previousframe using template matching and then obtains the position of a regionthat is most similar to the region of the reference image 251, as shownon the right side of FIG. 22. Then, the flight instruction generationunit 232 computes a movement amount from the obtained position of theregion to the target position, thereby computing a flight controlcommand again, and then transmits the command to the flying device 100.This process is repeated until the region of the reference image 251 (orthe region that is most similar to the region of the reference image251) is sufficiently close to the target position.

During the drag operation on the display unit 210, the flightinstruction generation unit 232 detects the moment at which the usertouches the display unit 210 with his or her finger, and then registersthe region of the predetermined size having the position that the usertouches with his or her finger as the center as the reference image 251.Then, the flight instruction generation unit 232 keeps updating aposition of the user's finger on the latest image as the target positionuntil the user separates his or her finger from the display unit 210.Then, when the user separates his or her finger from the display unit210, the flight instruction generation unit 232 sets the last positionthat the user touches with his or her finger as the target position asit is.

The flight instruction generation unit 232 can execute search for theimage that is most similar to the reference image 251 by changing asearch range (scale). Even if the size of the region of the referenceimage 251 is changed, the flight instruction generation unit 232 cancompute a movement amount from a current size to a target size as thecontrol of the position and convert the amount into a flight controlcommand by executing search for the image with the changed scale.

The controller 200 can cause the flight control command to be generatedbased on the information designated by the user by touching the displayunit 210, thereby controlling a flight state of the flying device 100.When the user newly touches another spot of the display unit 210 whilethe controller 200 controls a flight state of the flying device 100based on a user operation, the controller can promptly replace a controltarget and then control the flight state of the flying device 100.

The correspondence relationship between a user operation on an imagecaptured by the imaging device 101 and a movement of the flying device100 will be described. FIGS. 23 to 26 are illustrative diagrams showingexamples of the correspondence relationship between a user operationperformed with respect to an image captured by an imaging device 101 anda movement of the flying device 100.

FIG. 23 shows the correspondence relationship between a user operationand a movement of the flying device 100 when the user executes anoperation of moving an image captured by the imaging device 101 in theleft direction. As shown in FIG. 23, when the user executes theoperation of moving the image captured by the imaging device 101 in theleft direction, the controller 200 causes a flight control command torotate the flying device 100 horizontally clockwise to be generated andthen causes the command to be transmitted to the flying device 100.

When the flying device 100 receives the flight control command to rotatethe flying device horizontally clockwise from the controller 200,rotation of the rotors 104 a to 104 d is controlled such that the flyingdevice flies while rotating horizontally clockwise as shown in FIG. 23.As the flying device 100 flies while rotating horizontally clockwise asdescribed above, the flying device can change a position and an attitudethereof in a position in which an image can be captured in a formationdesired by a user.

FIG. 24 shows the correspondence relationship between a user operationand a movement of the flying device 100 when the user executes anoperation of moving an image captured by the imaging device 101 in thedownward direction. As shown in FIG. 24, when the user executes theoperation of moving the image captured by the imaging device 101 in thedownward direction, the controller 200 causes a flight control commandto raise the flying device 100 to be generated and then causes thecommand to be transmitted to the flying device 100.

When the flying device 100 receives the flight control command to raisethe flying device from the controller 200, rotation of the rotors 104 ato 104 d is controlled such that the flying device flies upward as shownin FIG. 24. By flying upward, the flying device 100 can change itsposition and attitude in a position in which an image can be captured ina formation desired by the user.

FIG. 25 shows the correspondence relationship between a user operationand a movement of the flying device 100 when the user executes a pinchoperation on an image captured by the imaging device 101 to reduce theimage. As shown in FIG. 25, when the user executes the pinch operationon the image captured by the imaging device 101 to reduce the image, thecontroller 200 causes a flight control command to cause the flyingdevice 100 to retreat to be generated and then causes the command to betransmitted to the flying device 100. Note that, herein, description isprovided on the assumption that the side on which the imaging device 101is provided is the front side of the flying device 100.

When the flying device 100 receives the flight control command to causethe flying device to retreat from the controller 200, the flying devicecontrols rotation of the rotors 104 a to 104 d to fly backward as shownin FIG. 25. By flying backward as described above, the flying device 100can change its position and attitude in a position at which an image canbe captured in a size desired by the user through the pinch operation.

FIG. 26 shows the correspondence relationship between a user operationand a movement of the flying device 100 when the user executes a pinchoperation on an image captured by the imaging device 101 to enlarge theimage. As shown in FIG. 26, when the user executes the pinch operationon the image captured by the imaging device 101 to enlarge the image,the controller 200 causes a flight control command to cause the flyingdevice 100 to advance to be generated and then causes the command to betransmitted to the flying device 100. Note that, herein, description isprovided on the assumption that the side on which the imaging device 101is provided is the frontside of the flying device 100.

When the flying device 100 receives the flight control command to causethe flying device to advance from the controller 200, the flying devicecontrols rotation of the rotors 104 a to 104 d to fly forward as shownin FIG. 26. By flying to advance as described above, the flying device100 can change a position and an attitude thereof in a position in whichan image can be captured in a size desired by the user through the pinchoperation.

As described above, the controller 200 can convert a user operationperformed on an image captured by the imaging device 101 into a flightcontrol command of the flying device 100, and then transmit the flightcontrol command to the flying device 100. In addition, the flying device100 can change a position and an attitude thereof according to the useroperation performed on the image captured by the imaging device 101.

As shown in FIG. 12, the flying device 100 includes the sensor unit 130.Since the flying device 100 includes the sensor unit 130, the flyingdevice can detect surrounding obstacles. For example, if the flyingdevice 100 includes an ultrasonic sensor as the sensor unit 130, theflying device 100 can estimate the distance from a floor to a ceilingduring its flight. In addition, if the flying device 100 includes animage sensor as the sensor unit 130, for example, the flying device 100can estimate the distance to a surrounding wall or the like based onenvironment recognition using the imaging device 101.

In addition, a case in which the flying device 100 comes into contactwith a surrounding obstacle when the flying device moves to create adesignated formation based on a user operation with respect to the imagecaptured by the imaging device 101 is considered. In such a case, thecontroller 200 may cause a movement limitation to be displayed on thedisplay unit 210 based on provision of information from the flyingdevice 100.

FIG. 27 is an illustrative diagram showing an example of informationdisplayed on the display unit 210 of the controller 200 constituting theimaging system 10 according to the embodiment of the present disclosure.FIG. 27 shows a state of a limitation range v17 that indicates alimitation of a sliding operation being displayed on the display unit210. When sliding is performed by designating the inner side of thelimitation range v17, the limitation range v17 notifies a user that theflying device 100 may collide with an obstacle such as a ceiling or thelike. The controller 200 can notify the user that the flying device 100may collide with an obstacle such as a ceiling by displaying thelimitation range v17 in a flickering manner.

In addition, the controller 200 may control display of the display unit210 to notify the user of a difficulty of changing a desired formationby causing the moving arrow v15 to flicker in a different color (forexample, red) from a normal one as shown in FIG. 27.

The imaging device 101 included in the flying device 100 can executeoptical zooming using movements of a zoom lens, digital zooming usingimage conversion, or zooming by actually approaching or retreating froma subject. When the imaging device 101 has such a zoom function and aformation is designated by the user performing the pinch operation asshown in FIG. 21, for example, the flying device 100 has difficultydetermining what kind of zoom function is better for zooming.

Thus, in general, the flying device 100 can preferentially execute theoptical zooming in order to avoid a risk of collision caused by amovement. However, since the optical zoom certainly has a limitation inits zoom range, when the formation designated by the user is difficultto create using only the optical zoom, the flying device 100 can firstoperate a zoom lens of the imaging device 101, and then fly toward oraway from a subject.

In addition, a case in which a shift occurs in a target change due toinfluence of wind or the like during flight control of the flying device100 as shown in FIG. 22 is considered. When such a shift occurs in thetarget size, the flying device 100 performs flight control so that theshift in the target size is corrected by a movement. When a position atwhich imaging is performed by the imaging device 101 is shifted, flightcontrol to correct the shift in the target size is controlled, therebypreventing an increasing shift from the original position withcorrection using zooming. In other words, the flying device 100 may makea change in a size using the optical zoom when the change in size isexplicitly designated as shown in FIG. 21, or may adjust its position bya movement for other controls.

In order to further facilitate flight control of the flying device 100,a flight control command to cause the flying device 100 to move only inone direction may be generated by the controller 200. For example, whena user is caused to tap the display unit 210 once in the examplesdescribed above, the controller 200 performs flying control of theflying device 100 using image feedback so that the tapped position ispositioned at the center of a captured image.

However, various kinds of operations such as those shown below may beprovided to the user to further facilitate flight control of the flyingdevice 100. For example, when the user is caused to tap the display unit210 once and then is caused to perform a drag operation from the tappedposition, the controller 200 can execute flight control such that theflying device 100 performs yaw-axis rotation.

FIG. 28 is an illustrative diagram showing an example of thecorrespondence relationship between a user operation performed withrespect to an image captured by the imaging device 101 and a movement ofthe flying device 100. FIG. 28 shows an example when the user is causedto tap the display unit 210 and then to perform the drag operation fromthe tapped position.

As shown in FIG. 28, based on three-dimensional positions of the imagingdevice 101 and a tapped object, a coordinate system in which the centerof the object is set to be the origin, the orientation of an opticalaxis of the imaging device is set to be the X axis, and two axesorthogonal to the X axis are set to be the Y and Z axes is obtained.When the user is caused to tap the display unit 210 once and then toperform the drag operation from the tapped position, the controller 200can execute flight control such that the flying device 100 performs theyaw-axis rotation about the Z axis in the coordinate system.Accordingly, manipulation of the flying device 100 becomes easier and animage is captured by the imaging device 101 when the flying device 100flies around the object, and the flying device 100 can capture the imagewith an effect of panoramic imaging.

In addition, when the user is caused to tap the display unit 210 onceand then to perform the pinch operation from the tapped position, thecontroller 200 can execute flight control such that the flying device100 flies toward or away from a subject.

FIG. 29 is an illustrative diagram showing another example of thecorrespondence relationship between a user operation performed withrespect to an image captured by the imaging device 101 and a movement ofthe flying device 100. FIG. 29 shows an example when the user is causedto tap the display unit 210 once and then to perform the pinch operationin the tapped position.

As shown in FIG. 29, when the user is caused to tap the display unit 210once and then to perform the pinch operation in the tapped position, thecontroller 200 can execute flight control such that the flying device100 flies forward or backward along the X axis. A coordinate system inthis case is decided based on three-dimensional positions of the imagingdevice 101 and a tapped object in the same manner as in FIG. 28. Bycapturing an image with the imaging device 101 while the flying device100 flies as described above, the flying device 100 can capture theimage of a subject in a size desired by the user.

As described above, the imaging system 10 according to the embodiment ofthe present disclosure causes the user to perform an operation on animage captured by the imaging device 101 included in the flying device100 through the controller 200 rather than direct manipulation of theflying device 100. Based on the operation performed with respect to theimage captured by the imaging device 101, the controller 200 causes aflight control command of the flying device 100 to be generated and thencontrols flight of the flying device 100 based on the generated flightcontrol command.

In this manner, by performing the operation on the image captured by theimaging device 101 included in the flying device 100 through thecontroller 200, the user can enjoy maneuvering the flying device 100even when the user is not skilled at maneuvering the flying device 100.In addition, by capturing an image using the imaging device 101 includedin the flying device 100, the user can easily capture an airborne image.

The imaging system 10 according to the embodiment of the presentdisclosure can allow a user to very easily maneuver the flying device100. On the other hand, when imaging is performed using the flyingdevice 100, the user does not have to hold a camera, and a situation inwhich it is difficult to control the flying device 100 as the userintends due to unexpected disturbance such as a sudden gust of wind orthe like is considered. When the flying device 100 is difficult tocontrol as the user intends, there may be a risk of collision with anenvironmental element such as a human, a wall, a ceiling, or the like.

The flying device 100 that constitutes the imaging system 10 accordingto the embodiment of the present disclosure includes the alert issuingunit 140 as shown in FIG. 12 in consideration of such a situation. Thealert issuing unit 140 generates an alert using a sound, light, or thelike based on control of the control unit 110 when the flying device 100attempts to fly over a pre-set flight range. When the flying device 100attempts to fly over the pre-set flight range, the alert issuing unit140 generates an alert and thereby the flying device 100 canautomatically warn the user or people nearby.

FIG. 30 is an illustrative diagram showing a state of the flying device100 positioned inside a predetermined flight range 21 that has aninitial position 20 as the center thereof. When unexpected disturbancesuch as a sudden gust of wind or the like occurs in this state, there isconcern of the flying device 100 flying beyond the flight range 21. FIG.31 is an illustrative diagram showing a state of the flying device 100attempting to fly beyond the predetermined flight range 21 that has theinitial position 20 as the center thereof.

Here, determination of whether or not the pre-set flight range has beenexceeded may be made by the control unit 110 comparing, for example,position information acquired by the position information acquisitionunit 132 included in the flying device 100 to position informationacquired by the position information acquisition unit 240 included inthe controller 200. When the determination is made based on thecomparison of the position information, periodic transmission andreception of the position information between the flying device 100 andthe controller 200 can be performed.

When an instruction is not transmitted from the controller 200manipulated by the user for another predetermined period even though thealert issuing unit 140 has generated an alert, the flying device 100 canautomatically land by controlling rotation of the motors 108 a to 108 dto cause rotation of the rotors 104 a to 104 d to slow down or stop.Determination of whether or not an instruction has been transmitted fromthe controller 200 after the alert issuing unit 140 has generated thealert can be made by the control unit 110. FIG. 32 is an illustrativediagram showing a state of the flying device 100 attempting automaticlanding when an instruction has not been transmitted from the controller200 even after the alert issuing unit 140 has generated the alert.

As described above, even when a situation in which it is difficult tocontrol the flying device 100 as the user intends occurs due to amaneuver error made by the user or unexpected disturbance such as asudden gust of wind or the like, the flying device 100 can warn peoplenearby by issuing an alert that the device has exceeded its pre-setflight range. In addition, when no instruction is given from thecontroller 200 even when an alert has been issued, the flying device 100can perform automatic landing in order to avoid a collision with aperson or an environmental element. With the operations as describedabove, the flying device 100 can drastically reduce a possibility of acollision with a person or an environmental element.

Since the flying device 100 includes the imaging device 101, the imagingsystem 10 according to the embodiment of the present disclosure canrealize an operation of causing the flying device 100 to take off from apalm of the user, performing imaging with the imaging device 101, andthen causing the flying device 100 to automatically return to the user.Hereinbelow, such an operation of the flying device 100 will bedescribed.

FIG. 33 is an illustrative diagram showing an overview of an operationof causing the flying device 100 to perform automatic takeoff from auser, imaging, and returning to the user using the imaging system 10according to the embodiment of the present disclosure. As shown in FIG.33, when the flying device 100 that is present in a user's hand at firststarts taking off based on, for example, a takeoff instruction from thecontroller 200, the flying device automatically flies to a position inwhich an appearance of the user can be imaged in pre-set formation.Then, after imaging of the user in the set formation is performed, theflying device 100 automatically returns to the user.

Since the flying device 100 flies as described above, the imaging system10 according to the embodiment of the present disclosure can capture animage similar to an image captured using a tripod using the flyingdevice 100 even in a place in which, for example, it is difficult to seta tripod.

There are various technologies for realizing automatic flying, imaging,and returning as described above, but in description below, a technologythat realizes automatic flying, imaging, and returning based on imagerecognition will be disclosed.

FIG. 34 is a flowchart showing an operation example of the imagingsystem 10 according to the embodiment of the present disclosure. Thedrawing of FIG. 34 is of the operation example of the imaging system 10when automatic flight, imaging, and returning of the flying device 100based on image recognition are executed. Hereinbelow, the operationexample of the imaging system 10 according to the embodiment of thepresent disclosure will be described using FIG. 34.

First, a user sets a desired formation in the imaging system 10 (StepS201). The imaging system 10 stores the formation set in Step S201 (StepS202). Various kinds of methods can be used in setting the formation,but an example of a method for setting the formation is as follows. Forexample, the imaging system 10 may allow the user to set the formationby allowing the user to touch the display unit 210 to designate adesired position and size of a face.

In addition, the imaging system 10 may cause the formation set in StepS201 to be stored in any of the flying device 100 or the controller 200.In addition, the imaging system 10 may cause the formation set in StepS201 to be stored in another device, for example, an external serverdevice connected to the Internet, rather than in any of the flyingdevice 100 or the controller 200.

When the desired formation is set in the imaging system 10, the userthen causes the imaging system 10 to memorize the face that is animaging target by capturing the image with the imaging device 101 (StepS203). The imaging system 10 creates a face recognition dictionary foridentifying faces using images of faces captured by the imaging device101 (Step S204).

The imaging system 10 may create the face recognition dictionary in anyof the flying device 100 or the controller 200. In addition, the imagingsystem 10 may cause the created face recognition dictionary to be storedin another device, for example, an external server device connected tothe Internet, rather than in any of the flying device 100 or thecontroller 200.

When the user has caused the imaging system 10 to memorize the face ofthe imaging target, the user then manipulates the controller 200 tocause the flying device 100 to take off (Step S205). The flying device100 that has taken off flies while capturing an image with the imagingdevice 101. Then, the flying device 100 flies while controlling itsposition so that the stored position and size of the face in the imagebeing captured by the imaging device 101 satisfy the formation set bythe user (Step S206).

The control of the position in Step S206 can be performed according to,for example, whether or not the stored face is included in the imagecaptured by the imaging device 101 or, if the stored image is included,the relationship between the position and the size of the face and aposition and a size designated by the user.

When the flying device 100 determines that the stored position and thesize of the face in the image being captured by the imaging device 101satisfy the formation set by the user, the imaging device 101 capturesthe image (Step S207).

When the image is captured in Step S207, the flying device 100 flieswhile performing position control so that the face stored in the imagebeing captured by the imaging device 101 is positioned at a shortdistance and on the front side thereof, in order to return to the user(Step S208). The position control in Step S208 can be performed in thesame manner as the position control in Step S206.

When the flying device 100 is in sufficient proximity to the user, theuser stretches out his or her palm under the flying device 100 (StepS209). When the flying device 100 detects stretching out of the user'spalm using the sensor unit 130, the flying device gradually slows downrotation of the rotors 104 a to 104 d, and lands on the user'soutstretched palm (Step S210).

With the operations of the flying device 100 and the controller 200described above, the imaging system 10 according to the embodiment ofthe present disclosure can realize automatic flight, imaging, andreturning of the flying device 100. By causing the flying device 100 tofly as described above, the imaging system 10 according to theembodiment of the present disclosure can exhibit the effect that animage similar to an image captured using a tripod can be captured usingthe flying device 100 even in a place in which, for example, it isdifficult to set a tripod.

FIGS. 35 to 38 are illustrative diagrams showing control examples ofpositions and attitudes of the flying device 100 based on imagerecognition. FIG. 35 shows a case in which a face position v21 stored inan image being captured by the imaging device 101 is set apart from aface position v22 set by a user along the X axis. When a coordinate ofthe face position v21 is set to be Xf and a coordinate of the faceposition v22 is set to be Xt, a movement amount V of the flying device100 per unit time is obtained with the following formula.

Vθ=Kθ×(Xf−Xt)

Wherein, Kθ is a coefficient.

FIG. 36 shows a case in which the face position v21 stored in the imagebeing captured by the imaging device 101 is set apart from the faceposition v22 set by the user along the Y axis. When a coordinate of theface position v21 is set to be Yf and a coordinate of the face positionv22 is set to be Yt, a movement amount Vy of the flying device 100 perunit time is obtained with the following formula.

Vy=Ky x(Yf−Yt)

Wherein, Ky is a coefficient.

FIG. 37 shows a case in which a face orientation in the face positionv21 stored in the image being captured by the imaging device 101 isdifferent from a face orientation set by the user. When the faceorientation in the face position v21 is set to be θf and the faceorientation set by the user is set to be θt, a movement amount Vx of theflying device 100 per unit time is obtained with the following formula.

Vx=Kx×(θf−θt)

Wherein, Kx is a coefficient.

FIG. 38 shows a case in which a face size in the face position v21stored in the image being captured by the imaging device 101 isdifferent from a face size in the face position v22 set by the user.When the face size in the face position v21 is set to be Syf and theface size set by the user is set to be Syt, a movement amount Vz of theflying device 100 per unit time is obtained with the following formula.

Vz=Kz×(Syf−Syt)

Wherein, Kz is a coefficient.

The imaging system 10 according to the embodiment of the presentdisclosure can control a position and an attitude of the flying device100 by recognizing an image being captured by the imaging device 101 asdescribed above.

Although the automatic returning of the flying device 100 based on imagerecognition is realized in the above-described examples, the presentdisclosure is not limited thereto. For example, by comparing theposition information acquired by the position information acquisitionunit 132 of the flying device 100 to the position information acquiredby the position information acquisition unit 240 of the controller 200,the flying device 100 may execute flight control so as to approach aposition of the controller 200. When returning of the flying device 100is realized based on determination made by comparing the positioninformation, periodic transmission and reception of the positioninformation can be performed between the flying device 100 and thecontroller 200.

2. CONCLUSION

According to the embodiment of the present disclosure described above,the controller 200 that can maneuver the flying device 100 that flieswith the plurality of rotors with simple operations is provided. Thecontroller 200 causes images captured by the imaging device 101 providedin the flying device 100 to be displayed and converts operationsperformed with respect to the images by the user into commands formaneuvering the flying device 100.

The controller 200 according to the embodiment of the present disclosureconverts the operations performed by the user with respect to the imagescaptured by the imaging device 101 into commands for maneuvering theflying device 100, and then transmits the commands to the flying device100. Accordingly, the controller 200 according to the embodiment of thepresent disclosure enables the user to maneuver the flying device 100with an instantaneous operation.

Although the imaging system 10 obtained by integrating the controller200 and the flying device 100 has been exemplified in the embodimentdescribed above, it is needless to say that the present disclosure isnot limited thereto. For example, even when it is difficult to integratethe controller 200 with the flying device 100, for example, asmartphone, a tablet-type mobile terminal, or the like may function asthe controller 200.

It is not necessary to perform each step of a process executed by eachdevice of the present specification in a time-series manner in the orderdescribed as a sequence diagram or a flowchart. For example, each stepof a process executed by each device may be performed in an orderdifferent from the order described as a flowchart, or may be performedin parallel.

In addition, a computer program for causing hardware such as a CPU, aROM, and a RAM installed in each device to exhibit the equivalentfunctions to those of each of the devices described above can also becreated. In addition, a storage medium in which such a computer programis stored can also be provided. In addition, by configuring each of thefunctional blocks shown in the functional block diagram to be hardware,a series of processes can also be realized using hardware.

Hereinabove, although the preferred embodiments of the presentdisclosure have been described in detail with reference to theaccompanying drawings, the present disclosure is not limited thereto. Itis obvious that a person who has general knowledge in the field of thetechnology to which the present disclosure belongs can devise variouskinds of modified examples or altered examples within the scope of thetechnical gist described in the claims, and it is understood that suchexamples surely belong to the technical scope of the present disclosureas well.

For example, although an image captured by the imaging device 101provided in the flying device 100 is displayed and an operationperformed by the user with respect to the image is converted into acommand for maneuvering the flying device 100 in the embodimentsdescribed above, the present disclosure is not limited thereto. When theimaging device 101 has a panning function or a tilting function, forexample, the controller 200 may convert an operation performed by a userwith respect to an image captured by the imaging device 101 into acommand for a panning operation or a tilting operation of the imagingdevice 101 and then transmit the command to the flying device 100.

In addition, the controller 200 may control the flying device 100 toenable capturing of an image desired by a user by combining a commandfor maneuvering the flying device 100 with a command for the panningoperation of the tilting operation of the imaging device 101 generatedfrom an operation performed by the user with respect to the imagecaptured by the imaging device 101.

Additionally, the present technology may also be configured as below:

(1) A control device including:

an image display unit configured to acquire, from a flying body, animage captured by an imaging device provided in the flying body and todisplay the image; and

a flight instruction generation unit configured to generate a flightinstruction for the flying body based on content of an operationperformed with respect to the image captured by the imaging device anddisplayed by the image display unit.

(2) The control device according to (1), wherein the flight instructiongeneration unit generates a flight instruction for the flying body basedon content of an operation performed with respect to the image displayunit displaying an image captured by the imaging device.(3) The control device according to (2), wherein the flight instructiongeneration unit generates a flight instruction for the flying body forcausing the imaging device to perform imaging in a formationcorresponding to the operation performed with respect to the imagedisplay unit.(4) The control device according to (3), wherein the flight instructiongeneration unit generates a flight instruction for the flying bodyinstructing that a predetermined range designated in the operationperformed with respect to the image display unit be positioned at acenter of an image captured by the imaging device.(5) The control device according to (3), wherein the flight instructiongeneration unit generates a flight instruction for the flying bodyinstructing that a predetermined range designated in the operationperformed with respect to the image display unit be positioned in aposition desired by a user in an image captured by the imaging device.(6) The control device according to (3), wherein the flight instructiongeneration unit generates a flight instruction for the flying bodyinstructing that a predetermined range designated in the operationperformed with respect to the image display unit have a size desired bya user in an image captured by the imaging device.(7) The control device according to any one of (3) to (6), furtherincluding:

a display control unit configured to control display of the imagedisplay unit,

wherein the display control unit causes the image display unit todisplay a route of moving until the flying body is in a positionenabling the imaging device to perform imaging in the formationcorresponding to the operation performed with respect to the imagedisplay unit.

(8) The control device according to (3), wherein, when the operationperformed with respect to the image display unit is an operation ofenlarging the image, the flight instruction generation unit generates aflight instruction instructing the flying body to advance.(9) The control device according to (8), wherein the operation ofenlarging the image is a pinch operation performed with respect to theimage display unit.(10) The control device according to (3), wherein, when the operationperformed with respect to the image display unit is an operation ofreducing the image, the flight instruction generation unit generates aflight instruction instructing the flying body to retreat.(11) The control device according to (10), wherein the operation ofreducing the image is a pinch operation performed with respect to theimage display unit.(12) The control device according to (3), wherein, when the operationperformed with respect to the image display unit is an operation ofcausing the image to slide, the flight instruction generation unitgenerates a flight instruction instructing the flight body to move in ahorizontal direction.(13) The control device according to (3), wherein, when the operationperformed with respect to the image display unit is an operation ofsetting a designated position at a center of an image, the flightinstruction generation unit generates a flight instruction instructingthe flying body to move so that the center of the image captured by theimaging device is in the designated position.(14) The control device according to (13), wherein the operation ofsetting the designated position at the center of the image is a tappingoperation performed with respect to the image display unit.(15) The control device according to (1), wherein the image display unitdisplays a path of the flying body to a target position while the flyingbody moves based on a flight instruction generated by the flightinstruction generation unit.(16) The control device according to (1), wherein the image display unitdisplays a limit range in which the flying body is capable of moving onthe image in a superimposed manner in an operation performed withrespect to the image display unit.(17) A control method including:

acquiring, from a flying body, an image captured by an imaging deviceprovided in the flying body and displaying the image; and

converting content of an operation performed with respect to the imagecaptured by the imaging device and displayed in the step of displayinginto a flight instruction for the flying body.

(18) A computer program causing a computer to execute:

acquiring, from a flying body, an image captured by an imaging deviceprovided in the flying body and displaying the image; and

converting content of an operation performed with respect to the imagecaptured by the imaging device and displayed in the step of displayinginto a flight instruction for the flying body.

What is claimed is:
 1. A method performed by a flying camera, the methodcomprising: driving rotors attached to a body of the flying camera so asto fly the flying camera; with circuitry of the flying camera,wirelessly transmitting to a control device an image captured with acamera of the flying camera so the control device displays the image ona touch panel display; responding to a touch operation performed on thetouch panel display to indicate a designated target in the image tocontrol capturing by the camera; and with the circuitry of the flyingcamera, automatically controlling a flight position of the flying cameraso that the designated target is in a center of the image on the touchpanel display.
 2. The method of claim 1 further comprising: capturing,by the camera, the image while the flying camera is flying.
 3. Themethod of claim 1, wherein the automatically controlling includescontrolling flying of the flying camera by controlling the rotors. 4.The method of claim 1 further comprising: capturing, by the camera, theimage when a predetermined condition is satisfied.
 5. The method ofclaim 1, wherein the automatically controlling includes automaticallycontrolling the flight position of the flying camera based on imagerecognition of the designated target in images captured by the camera.6. The method of claim 5, further comprising: performing, in the imagescaptured by the camera, the image recognition of the designated targetto identify the designated target in the images captured by the camera.7. The method of claim 1, wherein the responding includes responding tothe touch operation performed on the touch panel display to indicate thedesignated target in the image to control capturing by the cameraanother image that contains the designated target.
 8. The method ofclaim 7, further comprising: transmitting the another image to thecontrol device to be displayed on the touch panel display.
 9. The methodof claim 1 further comprising: automatically controlling the touch paneldisplay to include an object frame to designate the designated targetaccording to a region in the image designated by the touch operation.10. The method of claim 9, wherein the object frame is displayed withinthe image displayed on the touch panel display.
 11. A flying camera,comprising: a body; rotors with arms attached to the body, the rotorsconfigured to be rotated respectively to fly the body; a camera disposedon the body and flown by the rotors with the body; and circuitryconfigured to wirelessly transmit to a control device an image capturedwith the camera so the control device displays the image on a touchpanel display, respond to a touch operation performed on the touch paneldisplay to indicate a designated target in the image to controlcapturing by the camera, and automatically control a flight position ofthe flying camera so that the designated target is in a center of theimage on the touch panel display.
 12. The flying camera of claim 11,wherein the camera captures the image while the flying camera is flying.13. The flying camera of claim 11, wherein the circuitry is configuredto automatically control the flight position by controlling flying ofthe flying camera by controlling the rotors.
 14. The flying camera ofclaim 11, wherein the camera captures the image when a predeterminedcondition is satisfied.
 15. The flying camera of claim 11, wherein thecircuitry is configured to automatically control the flight positionbased on image recognition of the designated target in images capturedby the camera.
 16. The flying camera of claim 15, wherein the circuitryis configured to perform, in the images captured by the camera, theimage recognition of the designated target to identify the designatedtarget in the images captured by the camera.
 17. The flying camera ofclaim 11, wherein the responding includes responding to the touchoperation performed on the touch panel display to indicate thedesignated target in the image to control capturing by the cameraanother image that contains the designated target.
 18. The flying cameraof claim 17, wherein the circuitry is configured to transmit the anotherimage to the control device to be displayed on the touch panel display.19. The flying camera of claim 11, wherein the circuitry is configuredto automatically control the touch panel display to include an objectframe to designate the designated target according to a region in theimage designated by the touch operation.
 20. The flying camera of claim19, wherein the object frame is displayed within the image displayed onthe touch panel display.